Sketching for Large-Scale Learning of Mixture Models

Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a "compressive learning" framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set, and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical Expectation-Maximization (EM) technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 10 8 training samples) for the task of model-based speaker verification. Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive information preservation guarantees, in the spirit of infinite-dimensional compressive sensing

Multimodal Kalman Filtering

International audienceA difficult aspect of multimodal estimation is the possible discrepancybetween the sampling rates and/or the noise levels of theconsidered data. Many algorithms cope with these dissimilaritiesempirically. In this paper, we propose a conceptual analysis ofmultimodality where we try to find the “optimal” way of combiningmodalities. More specifically, we consider a simple Kalman filteringframework where several noisy sensors with different samplingfrequences and noise variances regularly observe a hidden state.We experimentally underline some relationships between the samplinggrids and the asymptotic variance of the maximum a posteriori(MAP) estimator. However, the explicit study of the asymptoticvariance seems intractable even in the simplest cases. We describe apromising idea to circumvent this difficulty: exploiting a stochasticmeasurement model for which one can more easily study the averageasymptotic behavior

Fundamental performance limits for ideal decoders in high-dimensional linear inverse problems

This paper focuses on characterizing the fundamental performance limits that can be expected from an ideal decoder given a general model, ie, a general subset of "simple" vectors of interest. First, we extend the so-called notion of instance optimality of a decoder to settings where one only wishes to reconstruct some part of the original high dimensional vector from a low-dimensional observation. This covers practical settings such as medical imaging of a region of interest, or audio source separation when one is only interested in estimating the contribution of a specific instrument to a musical recording. We define instance optimality relatively to a model much beyond the traditional framework of sparse recovery, and characterize the existence of an instance optimal decoder in terms of joint properties of the model and the considered linear operator. Noiseless and noise-robust settings are both considered. We show somewhat surprisingly that the existence of noise-aware instance optimal decoders for all noise levels implies the existence of a noise-blind decoder. A consequence of our results is that for models that are rich enough to contain an orthonormal basis, the existence of an L2/L2 instance optimal decoder is only possible when the linear operator is not substantially dimension-reducing. This covers well-known cases (sparse vectors, low-rank matrices) as well as a number of seemingly new situations (structured sparsity and sparse inverse covariance matrices for instance). We exhibit an operator-dependent norm which, under a model-specific generalization of the Restricted Isometry Property (RIP), always yields a feasible instance optimality property. This norm can be upper bounded by an atomic norm relative to the considered model.Comment: To appear in IEEE Transactions on Information Theor

Échantillonnage compressé et réduction de dimension pour l'apprentissage non supervisé

Cette thèse est motivée par la perspective de rapprochement entre traitement du signal et apprentissage statistique, et plus particulièrement par l'exploitation de techniques d'échantillonnage compressé afin de réduire le coût de tâches d'apprentissage. Après avoir rappelé les bases de l'échantillonnage compressé et mentionné quelques techniques d'analyse de données s'appuyant sur des idées similaires, nous proposons un cadre de travail pour l'estimation de paramètres de mélange de densités de probabilité dans lequel les données d'entraînement sont compressées en une représentation de taille fixe. Nous instancions ce cadre sur un modèle de mélange de Gaussiennes isotropes. Cette preuve de concept suggère l'existence de garanties théoriques de reconstruction d'un signal pour des modèles allant au-delà du modèle parcimonieux usuel de vecteurs. Nous étudions ainsi dans un second temps la généralisation de résultats de stabilité de problèmes inverses linéaires à des modèles tout à fait généraux de signaux. Nous proposons des conditions sous lesquelles des garanties de reconstruction peuvent être données dans un cadre général. Enfin, nous nous penchons sur un problème de recherche approchée de plus proche voisin avec calcul de signature des vecteurs afin de réduire la complexité. Dans le cadre où la distance d'intérêt dérive d'un noyau de Mercer, nous proposons de combiner un plongement explicite des données suivi d'un calcul de signatures, ce qui aboutit notamment à une recherche approchée plus précise.This thesis is motivated by the perspective of connecting compressed sensing and machine learning, and more particularly by the exploitation of compressed sensing techniques to reduce the cost of learning tasks. After a reminder of compressed sensing and a quick description of data analysis techniques in which similar ideas are exploited, we propose a framework for estimating probability density mixture parameters in which the training data is compressed into a fixed-size representation. We instantiate this framework on an isotropic Gaussian mixture model. This proof of concept suggests the existence of theoretical guarantees for reconstructing signals belonging to models beyond usual sparse models. We therefore study generalizations of stability results for linear inverse problems for very general models of signals. We propose conditions under which reconstruction guarantees can be given in a general framework. Finally, we consider an approximate nearest neighbor search problem exploiting signatures of the database vectors in order to save resources during the search step. In the case where the considered distance derives from a Mercer kernel, we propose to combine an explicit embedding of data followed by a signature computation step, which principally leads to a more accurate approximate search

Compressed sensing and dimensionality reduction for unsupervised learning

This thesis is motivated by the perspective of connecting compressed sensing and machine learning, and more particularly by the exploitation of compressed sensing techniques to reduce the cost of learning tasks. After a reminder of compressed sensing and a quick description of data analysis techniques in which similar ideas are exploited, we propose a framework for estimating probability density mixture parameters in which the training data is compressed into a fixed-size representation. We instantiate this framework on an isotropic Gaussian mixture model. This proof of concept suggests the existence of theoretical guarantees for reconstructing signals belonging to models beyond usual sparse models. We therefore study generalizations of stability results for linear inverse problems for very general models of signals. We propose conditions under which reconstruction guarantees can be given in a general framework. Finally, we consider an approximate nearest neighbor search problem exploiting signatures of the database vectors in order to save resources during the search step. In the case where the considered distance derives from a Mercer kernel, we propose to combine an explicit embedding of data followed by a signature computation step, which principally leads to a more accurate approximate search.Cette thèse est motivée par la perspective de rapprochement entre traitement du signal et apprentissage statistique, et plus particulièrement par l'exploitation de techniques d'échantillonnage compressé afin de réduire le coût de tâches d'apprentissage. Après avoir rappelé les bases de l'échantillonnage compressé et mentionné quelques techniques d'analyse de données s'appuyant sur des idées similaires, nous proposons un cadre de travail pour l'estimation de paramètres de mélange de densités de probabilité dans lequel les données d'entraînement sont compressées en une représentation de taille fixe. Nous instancions ce cadre sur un modèle de mélange de Gaussiennes isotropes. Cette preuve de concept suggère l'existence de garanties théoriques de reconstruction d'un signal pour des modèles allant au-delà du modèle parcimonieux usuel de vecteurs. Nous étudions ainsi dans un second temps la généralisation de résultats de stabilité de problèmes inverses linéaires à des modèles tout à fait généraux de signaux. Nous proposons des conditions sous lesquelles des garanties de reconstruction peuvent être données dans un cadre général. Enfin, nous nous penchons sur un problème de recherche approchée de plus proche voisin avec calcul de signature des vecteurs afin de réduire la complexité. Dans le cadre où la distance d'intérêt dérive d'un noyau de Mercer, nous proposons de combiner un plongement explicite des données suivi d'un calcul de signatures, ce qui aboutit notamment à une recherche approchée plus précise

Estimation de mélange de Gaussiennes sur données compressées

National audienceEstimating a probability mixture model from a set of vectors typically requires a large amount of memory if the data is voluminous. We propose a framework where the data is jointly compressed to a fixed-size representation called sketch, composed of empirical moments calculated from the data. By analogy with compressive sensing, we derive a parameter estimation algorithm from the sketch. We experimentally show that our algorithm allows precise estimation while consuming less memory than an EM algorithm for voluminous data. The algorithm also provides a privacy-preserving estimation tool since the sketch does not disclose information about individual datum it is based on

Compressive Gaussian Mixture Estimation

International audienceWe propose a framework to estimate the parameters of a mixture of isotropic Gaussians using empirical data drawn from this mixture. The difference with standard methods is that we only use a sketch computed from the data instead of the data itself. The sketch is composed of empirical moments computed in one pass on the data. To estimate the mixture parameters from the sketch, we derive an algorithm by analogy with Iterative Hard Thresholding, used in compressed sensing to recover sparse signals from a few linear projections. We prove experimentally that the parameters can be precisely estimated if the sketch is large enough, while using less memory than an EM algorithm if the data is numerous. Our approach also preserves the privacy of the initial data, since the sketch doesn't provide information on the individual data

Compressive Gaussian Mixture Estimation

International audienceWhen performing a learning task on voluminous data, memory and computational time can become prohibitive. In this chapter, we propose a framework aimed at estimating the parameters of a density mixture on training data in a compressive manner by computing a low-dimensional sketch of the data. The sketch represents empirical moments of the underlying probability distribution. Instantiating the framework on the case where the densities are isotropic Gaussians, we derive a reconstruction algorithm by analogy with compressed sensing. We experimentally show that it is possible to precisely estimate the mixture parameters provided that the sketch is large enough, while consuming less memory in the case of numerous data. The considered framework also provides a privacy-preserving data analysis tool, since the sketch does not disclose information about individual datum it is based on